{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HTML解析入门及准备URL生成连续技\n",
    "\n",
    "*  本周主要内容：批量抓取页面基础及技巧\n",
    "*  上周主要内容：HTML解析（parse HTML）及准备URL生成连续技\n",
    "*  20春_Web数据挖掘_week04\n",
    "*  电子讲义设计者：廖汉腾, 许智超\n",
    "<br/>\n",
    "<br/>\n",
    "\n",
    "-----\n",
    "## 复习\n",
    "\n",
    "复习：上周内容，实践\n",
    "\n",
    "* 翻页：参数字典的拆解\n",
    "* 翻页：系统性迭代\n",
    "* 翻页：数据备份与整合\n",
    "\n",
    "-----\n",
    "## 本周内容及学习目标\n",
    "\n",
    "搜狗微信：搜索关键字获取微信公众号相关文章\n",
    "  \n",
    "### 目标\n",
    "1. 使用 requests-html 爬取并存取网页文字档，查找[requests-html 中文文档](https://cncert.github.io/requests-html-doc-cn/#/)\n",
    "2. 熟悉 [xpath 语法](https://www.w3cschool.cn/xpath/xpath-syntax.html)丶[xpath 节点](https://www.w3cschool.cn/xpath/xpath-nodes.html)\n",
    "3. 使用 [xpath cheatsheet](https://devhints.io/xpath)\n",
    "  * 在 Chrome Inspector 使用\n",
    "  * 在 requests-html (Python) 使用\n",
    "4. 简易使用 [pd.DataFrame](https://www.pypandas.cn/doc/getting_started/dsintro.html#dataframe)\n",
    "5. 参数字典的拆解与迭代\n",
    "6. 翻页数据备份与整合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "/* 本电子讲义使用之CSS */\n",
       "div.code_cell {\n",
       "    background-color: #e5f1fe;\n",
       "}\n",
       "div.cell.selected {\n",
       "    background-color: #effee2;\n",
       "    font-size: 2rem;\n",
       "    line-height: 2.4rem;\n",
       "}\n",
       "div.cell.selected .rendered_html table {\n",
       "    font-size: 2rem !important;\n",
       "    line-height: 2.4rem !important;\n",
       "}\n",
       ".rendered_html pre code {\n",
       "    background-color: #C4E4ff;   \n",
       "    padding: 2px 25px;\n",
       "}\n",
       ".rendered_html pre {\n",
       "    background-color: #99c9ff;\n",
       "}\n",
       "div.code_cell .CodeMirror {\n",
       "    font-size: 2rem !important;\n",
       "    line-height: 2.4rem !important;\n",
       "}\n",
       ".rendered_html img, .rendered_html svg {\n",
       "    max-width: 60%;\n",
       "    height: auto;\n",
       "    float: right;\n",
       "}\n",
       "\n",
       ".rendered_html img[src*=\"#full\"], .rendered_html svg[src*=\"#full\"] {\n",
       "    max-width: 100%;\n",
       "    height: auto;\n",
       "    float: none;\n",
       "}\n",
       "\n",
       ".rendered_html img[src*=\"#thumbnail\"], .rendered_html svg[src*=\"#thumbnail\"] {\n",
       "    max-width: 15%;\n",
       "    height: auto;\n",
       "}\n",
       "\n",
       "/* Gradient transparent - color - transparent */\n",
       "hr {\n",
       "    border: 0;\n",
       "    border-bottom: 1px dashed #ccc;\n",
       "}\n",
       ".emoticon{\n",
       "    font-size: 5rem;\n",
       "    line-height: 4.4rem;\n",
       "    text-align: center;\n",
       "    vertical-align: middle;\n",
       "}\n",
       ".bg-split_apply_comine {\n",
       "    width: 500px;     \n",
       "    height: 300px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -10px -10px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-comine {\n",
       "    width: 175px;\n",
       "    height: 150px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -280px -80px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-apply {\n",
       "    width: 155px;\n",
       "    height: 225px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -160px -30px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-split {\n",
       "    width: 205px;\n",
       "    height: 225px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -10px -30px;\n",
       "    float: right;\n",
       "}\n",
       ".break {\n",
       "                   page-break-after: right; \n",
       "                   width:700px;\n",
       "                   clear:both;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>\n",
    "/* 本电子讲义使用之CSS */\n",
    "div.code_cell {\n",
    "    background-color: #e5f1fe;\n",
    "}\n",
    "div.cell.selected {\n",
    "    background-color: #effee2;\n",
    "    font-size: 2rem;\n",
    "    line-height: 2.4rem;\n",
    "}\n",
    "div.cell.selected .rendered_html table {\n",
    "    font-size: 2rem !important;\n",
    "    line-height: 2.4rem !important;\n",
    "}\n",
    ".rendered_html pre code {\n",
    "    background-color: #C4E4ff;   \n",
    "    padding: 2px 25px;\n",
    "}\n",
    ".rendered_html pre {\n",
    "    background-color: #99c9ff;\n",
    "}\n",
    "div.code_cell .CodeMirror {\n",
    "    font-size: 2rem !important;\n",
    "    line-height: 2.4rem !important;\n",
    "}\n",
    ".rendered_html img, .rendered_html svg {\n",
    "    max-width: 60%;\n",
    "    height: auto;\n",
    "    float: right;\n",
    "}\n",
    "\n",
    ".rendered_html img[src*=\"#full\"], .rendered_html svg[src*=\"#full\"] {\n",
    "    max-width: 100%;\n",
    "    height: auto;\n",
    "    float: none;\n",
    "}\n",
    "\n",
    ".rendered_html img[src*=\"#thumbnail\"], .rendered_html svg[src*=\"#thumbnail\"] {\n",
    "    max-width: 15%;\n",
    "    height: auto;\n",
    "}\n",
    "\n",
    "/* Gradient transparent - color - transparent */\n",
    "hr {\n",
    "    border: 0;\n",
    "    border-bottom: 1px dashed #ccc;\n",
    "}\n",
    ".emoticon{\n",
    "    font-size: 5rem;\n",
    "    line-height: 4.4rem;\n",
    "    text-align: center;\n",
    "    vertical-align: middle;\n",
    "}\n",
    ".bg-split_apply_comine {\n",
    "    width: 500px;     \n",
    "    height: 300px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -10px -10px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-comine {\n",
    "    width: 175px;\n",
    "    height: 150px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -280px -80px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-apply {\n",
    "    width: 155px;\n",
    "    height: 225px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -160px -30px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-split {\n",
    "    width: 205px;\n",
    "    height: 225px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -10px -30px;\n",
    "    float: right;\n",
    "}\n",
    ".break {\n",
    "                   page-break-after: right; \n",
    "                   width:700px;\n",
    "                   clear:both;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 基本模块\n",
    "import pandas as pd\n",
    "from requests_html import HTMLSession\n",
    "from pandas.api.types import is_numeric_dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 搜狗微信：文章搜索(模仿liepin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'e' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-3-2f9eb3cb36f0>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     52\u001b[0m \u001b[0m数据字典\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m{\u001b[0m\u001b[0mk\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mget_e_text\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mv\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdict_xpaths\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'text'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     53\u001b[0m \u001b[1;31m# href\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 54\u001b[1;33m \u001b[0m数据字典\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m{\u001b[0m\u001b[0mk\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mget_e_href\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mv\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdict_xpaths\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'href'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     55\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     56\u001b[0m \u001b[0m数据\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m数据字典\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-3-2f9eb3cb36f0>\u001b[0m in \u001b[0;36m<dictcomp>\u001b[1;34m(.0)\u001b[0m\n\u001b[0;32m     52\u001b[0m \u001b[0m数据字典\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m{\u001b[0m\u001b[0mk\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mget_e_text\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mv\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdict_xpaths\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'text'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     53\u001b[0m \u001b[1;31m# href\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 54\u001b[1;33m \u001b[0m数据字典\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m{\u001b[0m\u001b[0mk\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mget_e_href\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mv\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdict_xpaths\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'href'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     55\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     56\u001b[0m \u001b[0m数据\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m数据字典\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-3-2f9eb3cb36f0>\u001b[0m in \u001b[0;36mget_e_href\u001b[1;34m(_xpath_)\u001b[0m\n\u001b[0;32m     42\u001b[0m     \u001b[1;31m# 高级列表推导\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     43\u001b[0m     暂存结果 = [\"https://weixin.sogou.com\"+item\\\n\u001b[1;32m---> 44\u001b[1;33m             \u001b[1;32mfor\u001b[0m \u001b[0mitem\u001b[0m \u001b[1;32min\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mxpath\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_xpath_\u001b[0m\u001b[1;33m)\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     45\u001b[0m             for e in 主要元素]\n\u001b[0;32m     46\u001b[0m     \u001b[1;32mreturn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m暂存结果\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'e' is not defined"
     ]
    }
   ],
   "source": [
    "# A-1 query 为关键字选取 以python为例\n",
    "\n",
    "url = \"https://weixin.sogou.com/weixin\"\n",
    "session = HTMLSession()\n",
    "payload = {\n",
    "    \"query\":\"UX\",\n",
    "    \"type\":\"2\", # 2 文章、1 公众号\n",
    "    \"page\":10  \n",
    "}\n",
    "\n",
    "r = session.get( url, params = payload, )\n",
    "\n",
    "\n",
    "主要元素 =  r.html.xpath( '//ul[@class=\"news-list\"]/li')\n",
    "\n",
    "dict_xpaths = {\n",
    "    'text':{\n",
    "        'source':'//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/a/text()',\n",
    "        '时间':'//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/@t',       \n",
    "    },\n",
    "    'text_content':{\n",
    "        '标题':'//div[contains(@class,\"txt-box\")]/h3/a',\n",
    "        '摘要':'//div[contains(@class,\"txt-box\")]/p'\n",
    "    },\n",
    "    'href':{\n",
    "        \"公众号链接\":'//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/a/@href',\n",
    "        \"文章链接\":'//div[contains(@class,\"txt-box\")]/h3/a/@href',\n",
    "    }\n",
    "}\n",
    "\n",
    "\n",
    "def get_e_text_content(_xpath_):\n",
    "    # 高级列表推导\n",
    "    暂存结果 = [e.xpath(_xpath_)[0].lxml.text_content() for e in 主要元素]\n",
    "    return(暂存结果)\n",
    "\n",
    "def get_e_text(_xpath_):\n",
    "    暂存结果 = [\"\".join([x.strip() if type(x) is str else x.text.strip() for x in e.xpath(_xpath_)]) for e in 主要元素]\n",
    "    return(暂存结果)\n",
    "\n",
    "def get_e_href(_xpath_):\n",
    "    # 高级列表推导\n",
    "    暂存结果 = [\"https://weixin.sogou.com\"+item\\\n",
    "            for item in e.xpath(_xpath_)\\\n",
    "            for e in 主要元素]\n",
    "    return(暂存结果)\n",
    "\n",
    "数据字典 = dict()\n",
    "# content\n",
    "数据字典 = {k:get_e_text_content(v) for k,v in dict_xpaths['text_content'].items()}\n",
    "# text\n",
    "数据字典.update({k:get_e_text(v) for k,v in dict_xpaths['text'].items()})\n",
    "# href\n",
    "数据字典.update({k:get_e_href(v) for k,v in dict_xpaths['href'].items()})\n",
    "\n",
    "数据 = pd.DataFrame(数据字典)\n",
    "数据 = 数据.assign(\n",
    "    格式化时间 = lambda x: pd.to_datetime(x[\"时间\"], unit='s').dt.strftime('%Y-%m-%d %H:%M:%S')\n",
    ")\n",
    "数据\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open (\"weixin_UX.html\", encoding = \"utf8\", mode = \"w\") as fp:\n",
    "    fp.write(r.html.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## xpath的具体操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 来源的xpath解析结果\n",
    "r.html.xpath('//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/a/text()')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 标题？1、em text()? + a/text()?\n",
    "# 某一条<li>信息,再进行上一步 if else\n",
    "r.html.xpath('//div[contains(@class,\"txt-box\")]/h3/a/text()')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r.html.xpath('//div[contains(@class,\"txt-box\")]/h3/a')[0].lxml.text_content()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'互联网的兴起使交互设计成为了热门专业,相信很多同学最初接触这个专业时都有很多困惑为什么交互设计师又叫UI/UX设计师?为什...'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.html.xpath('//div[contains(@class,\"txt-box\")]/p')[0].lxml.text_content()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "can only concatenate str (not \"list\") to str",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-5-0a1457d43605>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# 公众号链接\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[1;34m'https://weixin.sogou.com/weixin'\u001b[0m\u001b[1;33m+\u001b[0m\u001b[0mr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mhtml\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mxpath\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/a/@href'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m: can only concatenate str (not \"list\") to str"
     ]
    }
   ],
   "source": [
    "# 公众号链接\n",
    "'https://weixin.sogou.com/weixin'+r.html.xpath('//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/a/@href')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 公众号文章链接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 时间"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 图片"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.1.2 Safari/603.3.8', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.request.headers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A-2 下载页面\n",
    "with open (\"weixin_用户研究.html\", encoding = \"utf8\", mode = \"w\") as fp:\n",
    "    fp.write(r.html.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'互联网的兴起使交互设计成为了热门专业,相信很多同学最初接触这个专业时都有很多困惑为什么交互设计师又叫UI/UX设计师?为什...'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lxml.text_content()\n",
    "r.html.xpath('//div[contains(@class,\"txt-box\")]/h3/a/em/text()')\n",
    "r.html.xpath('//div[contains(@class,\"txt-box\")]/h3/a/text()')\n",
    "r.html.xpath('//div[contains(@class,\"txt-box\")]/p')[0].lxml.text_content()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6EzDJysI4ql5MPrOUp16838dGRMI7NnPqGo6si9pJymdDP80tlD3nQwwvDqyjOWdzvlH_a5r65klUuXc2en3Gq_QQkGxcvz01GPlrCXgVybgzbjVvui9cEmt8NBnZRMT3pMkZ85Vs_1kiRUUmb8I5eEBLOCoQ4AMO6kc1fXLIZE31oCL3Fk9emG4OZCnxixtX&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " '/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6EzDJysI4ql5MPrOUp16838dGRMI7NnPqGo6si9pJymdDP80tlD3nQwwvDqyjOWdz-dY37aP99jh0cgB07tZomTs4n0O3d5wsEUqt-I-fg56kzAqGa_HVEKyfqWo2n-5FBxGTKiyloJoGna2dyuHLekB2mZY2qr5whozQbpD2FDT0P65pfc89LKUiiWOv1GPM&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " '/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6EzDJysI4ql5MPrOUp16838dGRMI7NnPqGo6si9pJymdDP80tlD3nQwwvDqyjOWdzyTcpXFbALUMDXrxlDu4ACaZ-0tHoZsNqxM98YuihUyW1c7TPu0OBx791bLKE-JHMVH-zu5ukyLKYPTNkaddccCPedhIIHhUjeoSI3uYFJgaUHFjZ3Wq0N2RTpLLTNgHY&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " '/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6EzDJysI4ql5MPrOUp16838dGRMI7NnPqGo6si9pJymdDP80tlD3nQwwvDqyjOWdzLu6p_YHPvqMomX8Jw7s4P98oVSgz3ujg8lI21epsCrqpdk52BWXqQqLvgu3IyWSg5ICIBmXgUSYtTKEfIeb4Uu-iMnGg9ajc6QVi6cXi0v2WzP4ptKeVceO00efWrWmm&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " '/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6EzDJysI4ql5MPrOUp16838dGRMI7NnPqGo6si9pJymdDP80tlD3nQwwvDqyjOWdzLu6p_YHPvqMomX8Jw7s4P98oVSgz3ujg8lI21epsCrqpdk52BWXqQqLvgu3IyWSg5ICIBmXgUSYtTKEfIeb4Uu-iMnGg9ajc6QVi6cXi0v2WzP4ptKeVceO00efWrWmm&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " '/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6EzDJysI4ql5MPrOUp16838dGRMI7NnPqGo6si9pJymdDP80tlD3nQwwvDqyjOWdzGrzojnwhR0AV6BSmwEA_qdSjvfqO8k2ouub5X0eJebSsbpdM95haE5YHZ6LgGtvAaJF7S645yNwIh6p6DG2_BavMWjxOXGciMAgCnAqA4n2mttLpfoCqu24OZCnxixtX&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " '/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6EzDJysI4ql5MPrOUp16838dGRMI7NnPqGo6si9pJymdDP80tlD3nQwwvDqyjOWdziNSiL1Avg2ao5rRQX6Hn9SdOk03IphjLaq2-I0yIEo0BfW3Gl-Ne4SHmLj5mnxePnYjhdLG3If-OCgmU-0ZzSxdYsfnETnzPF3CE6P8trv91vfj7tzLSj24OZCnxixtX&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " '/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6EzDJysI4ql5MPrOUp16838dGRMI7NnPqGo6si9pJymdDP80tlD3nQwwvDqyjOWdz2wB3jvhVko_T8ohFL1tGOQh9YzFn6dUMO96SkGcuP9o8DGuuar5-2HRqwvn2cWiqlktbSdI8BGrQDh88nrnM3aj55MwHUaoaGtkDMTCU_jgO-fWwnpkDT-O00efWrWmm&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " '/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6EzDJysI4ql5MPrOUp16838dGRMI7NnPqGo6si9pJymdDP80tlD3nQwwvDqyjOWdzXMqhETrsdE_RtAPw38mJvPrk99_X7i32HVbaTBmFvVXWokMAVF8SQl22rqYF0afKHpcUOJLPNyGnQg_KWVJ0E2ytbgVH0Rjli86bc4BcmBXzD1vWoFPaW2RTpLLTNgHY&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " '/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6EzDJysI4ql5MPrOUp16838dGRMI7NnPqGo6si9pJymdDP80tlD3nQwwvDqyjOWdzd8w_aBGyrdDMdhZ06333pbkkKpO8LNnbiEJwA_cW5xILqTSwey0_vK_u_elOboRfIvjX-DV_CXrb6bFIfvVI75t1vSAuhuec-Xw7wHMgG6fdJhPvKIC0HmRTpLLTNgHY&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.html.xpath('//div[contains(@class,\"txt-box\")]/h3/a/@href')\n",
    "r.html.xpath('//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/a/@href')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.1.2 Safari/603.3.8', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.request.headers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 下载页面和输出excel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A-2 下载页面\n",
    "with open (\"weixin_python.html\", encoding = \"utf8\", mode = \"w\") as fp:\n",
    "    fp.write(r.html.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name '数据' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-15-6b03e3af8a49>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# A-3 输出excel档\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0m数据\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mto_excel\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"20春_Web数据挖掘_week05_sougou_weixin.xlsx\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0msheet_name\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m\"搜查结果\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name '数据' is not defined"
     ]
    }
   ],
   "source": [
    "# A-3 输出excel档\n",
    "数据.to_excel(\"20春_Web数据挖掘_week05_sougou_weixin.xlsx\", sheet_name=\"搜查结果\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 拆分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 文章链接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://weixin.sogou.com/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS8HGqhNZfMkjyRECZNHHRwU4gV8M3bpyKVqXa8Fplpd9VSAQgPQHoz-cuiTWuZoKIhZ-p4N9Dd8LlsV1XIYolujTns9lFmG9q8K_VtY4tJDyrU-IbxSeqg80jztYGvgdM3fcImJM5DLJKHe_pvcfsO2FuZ71mefD4XitNXUKHRmE06lEpHilonChZ_CQK6aBKuXf8yUB81er1BrDz3FEksHyPfCoem7FzA..&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " 'https://weixin.sogou.com/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS8HGqhNZfMkjyRECZNHHRwU4gV8M3bpyKVqXa8Fplpd9sn8qHQ6YMaeuO1ceSJODfIrShJHplcanGwG2sEafoFLqrmU3eCW1AFVAiUSGHO056D932hvLaZgcIYt8YXRO9bCwN00Zmp5bWejTWiriLeXYELfnzf_nxVhoygh61ibqpnWmrnQSymLKkuBP39hU-6p_1GLKMtewNGslnKh3Rn4C2Qmj7RgGwQ..&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " 'https://weixin.sogou.com/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS8HGqhNZfMkjyRECZNHHRwU4gV8M3bpyKVqXa8Fplpd9AnCw0G9Va60zVew4C02Upyds6XvhoaxBI2YE0u2Cyv6uTQCmRHdOAKXRNHwlqkdEPZbWhzfcI0n2SHQrnqlOhoRGv2pna8rXbcok34_Id878MvurCsB46AgU5ALspd_8c4UUYtAhdmRUx99xdpQZectvonWpR2Nz6h9UZTyZPlX0OK9DLfgmPA..&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " 'https://weixin.sogou.com/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS8HGqhNZfMkjyRECZNHHRwU4gV8M3bpyKVqXa8Fplpd9nQG-qqOT6KgbFojtV7ikNLs298u3Phy7I8OtXGDSVpX06XNPQJm8AAXrG6a-23xHdKLJmMxm1j7aYLEcbCKv79qUl-SWb9R4IH1Cjg28n302aOPeSq4S0-sR_0gGLCMtR2954FhJfE5Cxc7Jw0iuIlZTXOkP021ERiOdls1XHSKTJmCU1UgHwQ..&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " 'https://weixin.sogou.com/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS8HGqhNZfMkjyRECZNHHRwU4gV8M3bpyKVqXa8Fplpd9nQG-qqOT6KgbFojtV7ikNLs298u3Phy7I8OtXGDSVpX06XNPQJm8ACPO3OfRzmYaKJLA6zKbE94uCeGTbdvYvc7L7qgldOlP2644yEhVxeifudmycMg1y1TsGYEtjp8l1fJjmObiuw8WA-_qsL9W_ButpDqLuN_DAUljDz6AItl6VKrzu_4XKA..&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " 'https://weixin.sogou.com/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS8HGqhNZfMkjyRECZNHHRwU4gV8M3bpyKVqXa8Fplpd9bQkojhPc9hQOEXWuM3XH0_QwB6xkootYsOSzbIAqM2YM-35FaZHsLulFQXTkr3MowJ7Fmb2pmgFEGHUUXU2LPiW3vno2ltnqoSpmtM3WXeL_-5vW0LDmhWqdFRGxeBzeSJpbOCnvWbcV6Ro2NuRh-latnhK4g5VdGXR3i5I4mfQO8fIRwtipOg..&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " 'https://weixin.sogou.com/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS8HGqhNZfMkjyRECZNHHRwU4gV8M3bpyKVqXa8Fplpd9ZC7VoXcKdAy3tjjcSB94GtZHKJYtIRUTKNmdJy-dr9D-ygb3ueiju9oIHad6FgaDSrPMbQ-TrlZHYhf4BCbgfUH6Kq6G5B1hCctVoCCPpbYIlYz5bXSIT7Ueou3EoO9o0Y3cx1Lppl41uu-dN7SXYXEKREC8-wu1PPMUhlus81Zr1dbZWbKUeQ..&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " 'https://weixin.sogou.com/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS8HGqhNZfMkjyRECZNHHRwU4gV8M3bpyKVqXa8Fplpd902Am6qzIPO260TfmoIDZ-Ik3PoyLL90Z3u3T1a8v0TsOFqv9A-PugdKQ2L98m8C_IydJIc9SrdFPWCk8DH85NX33XQqnQ0JFBjiM_qDwNZwxam6-SZtRbJp8PYfrQYBi3rFhCla2LqVSRUovrddfWkrdtY5ciP_QiAfEWixg6I7m0CtGSSkZQQ..&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " 'https://weixin.sogou.com/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS8HGqhNZfMkjyRECZNHHRwU4gV8M3bpyKVqXa8Fplpd9y_zbIRWomcP5ET7OQSgzK4ZpEc1NqOXAEz_HOkjvQA8lQh3QHptarLNykDjCIuFlOCNwUbZ7hvpw-sOwWfKE5UqECd6c38AtUvMF5o4R-XEUHeBs-7ke5dAgFbrVdefpDxYMjqssQz2jk4xRAUX6219RnUiLPp7x-WU2A4HdjwlFkDr8IT4KYw..&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C',\n",
       " 'https://weixin.sogou.com/link?url=dn9a_-gY295K0Rci_xozVXfdMkSQTLW6cwJThYulHEtVjXrGTiVgS8HGqhNZfMkjyRECZNHHRwU4gV8M3bpyKVqXa8Fplpd9DKfyhj-cJvpCnh3mPXVaH4-b9pWBo9v9tOmlIKD8I8V8z-16dExWt5ldqyrD0frxkVvNT_ydeiPg6-KL5Rl5SReJFtGuA91_IO5Zda0SqpQD7mfPVoCsbW5duRq70KBpGQmU99czmO4zR4cstFDglXxQrrfWeIXBtwt-ik5Vw19f_71bqkOtiw..&type=2&query=UX&token=3B01492A8E15414B3B3998817B8912983C69B4815E9DBA9C']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A-4 文章链接 细化\n",
    "# href_list 文章链接\n",
    "article_href_list = []\n",
    "for item in r.html.xpath('//div[contains(@class,\"txt-box\")]/h3/a/@href'):\n",
    "#     print(item)\n",
    "    article_href_list.append(\"https://weixin.sogou.com\"+item)\n",
    "article_href_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 公众号链接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'e' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-17-53e9601d5105>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# A-5 公众号链接 细化\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m [\"http://weixin.sogou.com\"+item\\\n\u001b[1;32m----> 3\u001b[1;33m  \u001b[1;32mfor\u001b[0m \u001b[0mitem\u001b[0m \u001b[1;32min\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mxpath\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/a/@href'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m  for e in 主要元素]\n",
      "\u001b[1;31mNameError\u001b[0m: name 'e' is not defined"
     ]
    }
   ],
   "source": [
    "# A-5 公众号链接 细化\n",
    "[\"http://weixin.sogou.com\"+item\\\n",
    " for item in e.xpath('//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/a/@href')\\\n",
    " for e in 主要元素]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 来源和发表时间"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A-6 来源和发表时间 细化\n",
    "source = r.html.xpath('//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/a/text()')\n",
    "time =  r.html.xpath('//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/@t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：参数字典的拆解\n",
    "\n",
    "## xpath解析翻页a/@href"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# B-0   单一页面\n",
    "url = \"https://weixin.sogou.com/weixin\"\n",
    "session = HTMLSession()\n",
    "payload = {\n",
    "    \"query\":\"python\",\n",
    "    \"type\":\"2\"\n",
    "}\n",
    "\n",
    "r = session.get( url, params = payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element 'a' href='?query=python&type=2&page=2&ie=utf8' id='sogou_page_2' uigs='page_2'>, <Element 'a' href='?query=python&type=2&page=3&ie=utf8' id='sogou_page_3' uigs='page_3'>, <Element 'a' href='?query=python&type=2&page=4&ie=utf8' id='sogou_page_4' uigs='page_4'>, <Element 'a' href='?query=python&type=2&page=5&ie=utf8' id='sogou_page_5' uigs='page_5'>, <Element 'a' href='?query=python&type=2&page=6&ie=utf8' id='sogou_page_6' uigs='page_6'>, <Element 'a' href='?query=python&type=2&page=7&ie=utf8' id='sogou_page_7' uigs='page_7'>, <Element 'a' href='?query=python&type=2&page=8&ie=utf8' id='sogou_page_8' uigs='page_8'>, <Element 'a' href='?query=python&type=2&page=9&ie=utf8' id='sogou_page_9' uigs='page_9'>, <Element 'a' href='?query=python&type=2&page=10&ie=utf8' id='sogou_page_10' uigs='page_10'>, <Element 'a' class=('np',) href='?query=python&type=2&page=2&ie=utf8' id='sogou_next' uigs='page_next'>]\n",
      "['?query=python&type=2&page=2&ie=utf8', '?query=python&type=2&page=3&ie=utf8', '?query=python&type=2&page=4&ie=utf8', '?query=python&type=2&page=5&ie=utf8', '?query=python&type=2&page=6&ie=utf8', '?query=python&type=2&page=7&ie=utf8', '?query=python&type=2&page=8&ie=utf8', '?query=python&type=2&page=9&ie=utf8', '?query=python&type=2&page=10&ie=utf8', '?query=python&type=2&page=2&ie=utf8']\n",
      "{'2': '?query=python&type=2&page=2&ie=utf8', '3': '?query=python&type=2&page=3&ie=utf8', '4': '?query=python&type=2&page=4&ie=utf8', '5': '?query=python&type=2&page=5&ie=utf8', '6': '?query=python&type=2&page=6&ie=utf8', '7': '?query=python&type=2&page=7&ie=utf8', '8': '?query=python&type=2&page=8&ie=utf8', '9': '?query=python&type=2&page=9&ie=utf8', '10': '?query=python&type=2&page=10&ie=utf8', '下一页': '?query=python&type=2&page=2&ie=utf8'}\n"
     ]
    }
   ],
   "source": [
    "# A-1  xpath 解析翻页a/@href\n",
    "xpath_翻页a = '//div[@class=\"p-fy\"]/a'\n",
    "xpath_翻页a = '//div[@class=\"p-fy\"]/a'\n",
    "print (r.html.xpath(xpath_翻页a))\n",
    "\n",
    "href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "print (href_列表)\n",
    "\n",
    "href_字典 = {x.text:x.xpath('//@href')[0]  for x in r.html.xpath(xpath_翻页a)}\n",
    "print (href_字典)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'parse_qs' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-21-3140a6cc88ee>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mparse_qs\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"query=python&type=2&page=2&ie=utf8\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m: name 'parse_qs' is not defined"
     ]
    }
   ],
   "source": [
    "parse_qs(\"query=python&type=2&page=2&ie=utf8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>scheme</th>\n",
       "      <th>netloc</th>\n",
       "      <th>path</th>\n",
       "      <th>params</th>\n",
       "      <th>query</th>\n",
       "      <th>fragment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>query=python&amp;type=2&amp;page=2&amp;ie=utf8</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>query=python&amp;type=2&amp;page=3&amp;ie=utf8</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>query=python&amp;type=2&amp;page=4&amp;ie=utf8</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>query=python&amp;type=2&amp;page=5&amp;ie=utf8</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>query=python&amp;type=2&amp;page=6&amp;ie=utf8</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>query=python&amp;type=2&amp;page=7&amp;ie=utf8</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>query=python&amp;type=2&amp;page=8&amp;ie=utf8</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>query=python&amp;type=2&amp;page=9&amp;ie=utf8</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>query=python&amp;type=2&amp;page=10&amp;ie=utf8</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>query=python&amp;type=2&amp;page=2&amp;ie=utf8</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  scheme netloc path params                                query fragment\n",
       "0                             query=python&type=2&page=2&ie=utf8         \n",
       "1                             query=python&type=2&page=3&ie=utf8         \n",
       "2                             query=python&type=2&page=4&ie=utf8         \n",
       "3                             query=python&type=2&page=5&ie=utf8         \n",
       "4                             query=python&type=2&page=6&ie=utf8         \n",
       "5                             query=python&type=2&page=7&ie=utf8         \n",
       "6                             query=python&type=2&page=8&ie=utf8         \n",
       "7                             query=python&type=2&page=9&ie=utf8         \n",
       "8                            query=python&type=2&page=10&ie=utf8         \n",
       "9                             query=python&type=2&page=2&ie=utf8         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scheme      1\n",
      "netloc      1\n",
      "path        1\n",
      "params      1\n",
      "query       9\n",
      "fragment    1\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ie</th>\n",
       "      <th>page</th>\n",
       "      <th>query</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>utf8</td>\n",
       "      <td>2</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>utf8</td>\n",
       "      <td>3</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>utf8</td>\n",
       "      <td>4</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>utf8</td>\n",
       "      <td>5</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>utf8</td>\n",
       "      <td>6</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>utf8</td>\n",
       "      <td>7</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>utf8</td>\n",
       "      <td>8</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>utf8</td>\n",
       "      <td>9</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>utf8</td>\n",
       "      <td>10</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>utf8</td>\n",
       "      <td>2</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     ie page   query type\n",
       "0  utf8    2  python    2\n",
       "1  utf8    3  python    2\n",
       "2  utf8    4  python    2\n",
       "3  utf8    5  python    2\n",
       "4  utf8    6  python    2\n",
       "5  utf8    7  python    2\n",
       "6  utf8    8  python    2\n",
       "7  utf8    9  python    2\n",
       "8  utf8   10  python    2\n",
       "9  utf8    2  python    2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ie       1\n",
      "page     9\n",
      "query    1\n",
      "type     1\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ie</th>\n",
       "      <th>page</th>\n",
       "      <th>query</th>\n",
       "      <th>type</th>\n",
       "      <th>page_int</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>utf8</td>\n",
       "      <td>2</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>utf8</td>\n",
       "      <td>3</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>utf8</td>\n",
       "      <td>4</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>utf8</td>\n",
       "      <td>5</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>utf8</td>\n",
       "      <td>6</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>utf8</td>\n",
       "      <td>7</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>utf8</td>\n",
       "      <td>8</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>utf8</td>\n",
       "      <td>9</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>utf8</td>\n",
       "      <td>10</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>utf8</td>\n",
       "      <td>2</td>\n",
       "      <td>python</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     ie page   query type  page_int\n",
       "0  utf8    2  python    2         2\n",
       "1  utf8    3  python    2         3\n",
       "2  utf8    4  python    2         4\n",
       "3  utf8    5  python    2         5\n",
       "4  utf8    6  python    2         6\n",
       "5  utf8    7  python    2         7\n",
       "6  utf8    8  python    2         8\n",
       "7  utf8    9  python    2         9\n",
       "8  utf8   10  python    2        10\n",
       "9  utf8    2  python    2         2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# A-2 建构参数模板：找到关键参数及参数结构\n",
    "\n",
    "# 需要模组库\n",
    "from urllib.parse import urlparse, parse_qs\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "\n",
    "df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "\n",
    "df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "\n",
    "display(df)\n",
    "\n",
    "\n",
    "\n",
    "print(df.nunique())\n",
    "display(df_qs)\n",
    "print(df_qs.nunique())\n",
    "\n",
    "df_qs.page\n",
    "df_qs = df_qs.assign (page_int=df_qs.page.astype(int)) # 变成整数(数值)\n",
    "display(df_qs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'query': ['python'], 'type': ['2'], 'page': ['4'], 'ie': ['utf8']}\n",
      "{'2': '?query=python&type=2&page=2&ie=utf8', '3': '?query=python&type=2&page=3&ie=utf8', '4': '?query=python&type=2&page=4&ie=utf8', '5': '?query=python&type=2&page=5&ie=utf8', '6': '?query=python&type=2&page=6&ie=utf8', '7': '?query=python&type=2&page=7&ie=utf8', '8': '?query=python&type=2&page=8&ie=utf8', '9': '?query=python&type=2&page=9&ie=utf8', '10': '?query=python&type=2&page=10&ie=utf8', '下一页': '?query=python&type=2&page=2&ie=utf8'}\n"
     ]
    }
   ],
   "source": [
    "# A-2 建构参数模板：找到关键参数及参数结构\n",
    "\n",
    "def parse_url_qs_for_page (url):\n",
    "    six_parts = urlparse(url) \n",
    "    out = parse_qs(six_parts.query)\n",
    "    return (out)\n",
    "\n",
    "# 取一例做模板\n",
    "参数模板 = parse_url_qs_for_page(href_列表[2])\n",
    "print (参数模板)\n",
    "\n",
    "# 重新审视：href_字典\n",
    "print (href_字典)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "10\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{1: {'query': ['用户体验'], 'type': ['2'], 'page': [1], 'ie': ['utf8']},\n",
       " 2: {'query': ['用户体验'], 'type': ['2'], 'page': [2], 'ie': ['utf8']},\n",
       " 3: {'query': ['用户体验'], 'type': ['2'], 'page': [3], 'ie': ['utf8']},\n",
       " 4: {'query': ['用户体验'], 'type': ['2'], 'page': [4], 'ie': ['utf8']},\n",
       " 5: {'query': ['用户体验'], 'type': ['2'], 'page': [5], 'ie': ['utf8']},\n",
       " 6: {'query': ['用户体验'], 'type': ['2'], 'page': [6], 'ie': ['utf8']},\n",
       " 7: {'query': ['用户体验'], 'type': ['2'], 'page': [7], 'ie': ['utf8']},\n",
       " 8: {'query': ['用户体验'], 'type': ['2'], 'page': [8], 'ie': ['utf8']},\n",
       " 9: {'query': ['用户体验'], 'type': ['2'], 'page': [9], 'ie': ['utf8']},\n",
       " 10: {'query': ['用户体验'], 'type': ['2'], 'page': [10], 'ie': ['utf8']}}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A-3 建构参数模板生成器：query（搜索关键字）page\n",
    "def 参数模板生成(query, page):\n",
    "    参数 = 参数模板.copy()\n",
    "    参数['page'] = page\n",
    "    参数['query'] = query\n",
    "    return (参数)\n",
    "\n",
    "# -----------    参数模版生成器 目的 构建payload\n",
    "# payload={\n",
    "    \n",
    "# }\n",
    "参数_query_用户体验_page = { \n",
    "    i:参数模板生成(page = [i], \\\n",
    "                  query = ['用户体验']) \\\n",
    "    for i,v in href_字典.items()\\\n",
    "    }\n",
    "\n",
    "print (df_qs.page_int.min()) # 最小值只有2 1的时候没有链接？\n",
    "print (df_qs.page_int.max()) # 最大值只有9\n",
    "\n",
    "参数_query_用户体验_page = { \n",
    "    i:参数模板生成(page = [i], \\\n",
    "                  query = ['用户体验']) \\\n",
    "    for i in range(df_qs.page_int.min()-1,df_qs.page_int.max()+1)\\\n",
    "    }\n",
    "参数_query_用户体验_page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 封装request_sougou_weixin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "第 1 页\n",
      "payload: {'query': 'python', 'type': '2', 'page': 1}\n",
      "url: https://weixin.sogou.com/weixin\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'list' object has no attribute 'sleep'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-25-ea64bf542d46>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     67\u001b[0m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"payload:\"\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mpayload\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     68\u001b[0m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"url:\"\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 69\u001b[1;33m     \u001b[0mtime\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m+\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0mrandom\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     70\u001b[0m     \u001b[0mdf\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrequest_sougou_weixin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mparams\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mpayload\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     71\u001b[0m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"第\"\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;34m\"页\"\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;34m\"的Dataframe\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mAttributeError\u001b[0m: 'list' object has no attribute 'sleep'"
     ]
    }
   ],
   "source": [
    "session = HTMLSession()\n",
    "def request_sougou_weixin(url,params):\n",
    "   \n",
    "    r = session.get( url, params = payload, )\n",
    "    \n",
    "    \n",
    "    主要元素 =  r.html.xpath( '//ul[@class=\"news-list\"]/li')\n",
    "\n",
    "    dict_xpaths = {\n",
    "        'text':{\n",
    "            'source':'//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/a/text()',\n",
    "            '时间':'//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/@t',       \n",
    "        },\n",
    "        'text_content':{\n",
    "            '标题':'//div[contains(@class,\"txt-box\")]/h3/a',\n",
    "            '摘要':'//div[contains(@class,\"txt-box\")]/p'\n",
    "        },\n",
    "        'href':{\n",
    "            \"公众号链接\":'//div[contains(@class,\"txt-box\")]/div[@class=\"s-p\"]/a/@href',\n",
    "            \"文章链接\":'//div[contains(@class,\"txt-box\")]/h3/a/@href',\n",
    "        }\n",
    "    }\n",
    "\n",
    "\n",
    "    def get_e_text_content(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [e.xpath(_xpath_)[0].lxml.text_content() for e in 主要元素]\n",
    "\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_text(_xpath_):\n",
    "        暂存结果 = [\"\".join([x.strip() if type(x) is str else x.text.strip() for x in e.xpath(_xpath_)]) for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_href(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [\"https://weixin.sogou.com\"+item\\\n",
    "                for item in e.xpath(_xpath_)\\\n",
    "                for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    数据字典 = dict()\n",
    "    数据字典 = {k:get_e_text_content(v) for k,v in dict_xpaths['text_content'].items()}\n",
    "\n",
    "    数据字典.update({k:get_e_text(v) for k,v in dict_xpaths['text'].items()})\n",
    "    数据字典.update({k:get_e_href(v) for k,v in dict_xpaths['href'].items()})\n",
    "\n",
    "    数据 = pd.DataFrame(数据字典)\n",
    "    数据 = 数据.assign(\n",
    "        格式化时间 = lambda x: pd.to_datetime(x[\"时间\"], unit='s').dt.strftime('%Y-%m-%d %H:%M:%S')\n",
    "    )\n",
    "    return(数据)\n",
    "\n",
    "\n",
    "url = \"https://weixin.sogou.com/weixin\"\n",
    "\n",
    "\n",
    "list_df=[]\n",
    "for i in range(1,10):\n",
    "    print(\"第\",i,\"页\")\n",
    "    payload = {\n",
    "        \"query\":\"python\",\n",
    "        \"type\":\"2\",\n",
    "        \"page\":i\n",
    "\n",
    "    } \n",
    "    print(\"payload:\",payload)\n",
    "    print(\"url:\",url)\n",
    "    time.sleep(10+10*random())\n",
    "    df = request_sougou_weixin(url,params=payload)\n",
    "    print(\"第\",i,\"页\",\"的Dataframe\")\n",
    "    display(df)\n",
    "    \n",
    "    df.to_csv(\"20春_Web数据挖掘_week05_sougou_weixin_{key}_{k}.tsv\"\\\n",
    "                  .format(key=\"python\", k=i), sep=\"\\t\", encoding=\"utf8\")\n",
    "        \n",
    "    df = df.assign (query = key)  # 区分  query   \n",
    "    df = df.assign (page = i)  # 区分  page    \n",
    "    list_df.append(df)\n",
    "    \n",
    "\n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week05_sougou_weixin_翻页_5.xlsx\",\\\n",
    "                sheet_name=\"_\".join(query))\n",
    "# 预估时间: 4*5秒*10 =200\n",
    "# 预估数量: 4*40*10 =1600"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<PreparedRequest [GET]>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 多个页面爬取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'query': ['python'], 'type': ['2'], 'page': ['1'], 'ie': ['utf8']}\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'e' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<timed exec>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-25-ea64bf542d46>\u001b[0m in \u001b[0;36mrequest_sougou_weixin\u001b[1;34m(url, params)\u001b[0m\n\u001b[0;32m     44\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     45\u001b[0m     \u001b[0m数据字典\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m{\u001b[0m\u001b[0mk\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mget_e_text\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mv\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdict_xpaths\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'text'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 46\u001b[1;33m     \u001b[0m数据字典\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m{\u001b[0m\u001b[0mk\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mget_e_href\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mv\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdict_xpaths\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'href'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     47\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     48\u001b[0m     \u001b[0m数据\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m数据字典\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-25-ea64bf542d46>\u001b[0m in \u001b[0;36m<dictcomp>\u001b[1;34m(.0)\u001b[0m\n\u001b[0;32m     44\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     45\u001b[0m     \u001b[0m数据字典\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m{\u001b[0m\u001b[0mk\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mget_e_text\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mv\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdict_xpaths\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'text'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 46\u001b[1;33m     \u001b[0m数据字典\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m{\u001b[0m\u001b[0mk\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mget_e_href\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mv\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mv\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mdict_xpaths\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'href'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     47\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     48\u001b[0m     \u001b[0m数据\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m数据字典\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-25-ea64bf542d46>\u001b[0m in \u001b[0;36mget_e_href\u001b[1;34m(_xpath_)\u001b[0m\n\u001b[0;32m     36\u001b[0m         \u001b[1;31m# 高级列表推导\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     37\u001b[0m         暂存结果 = [\"https://weixin.sogou.com\"+item\\\n\u001b[1;32m---> 38\u001b[1;33m                 \u001b[1;32mfor\u001b[0m \u001b[0mitem\u001b[0m \u001b[1;32min\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mxpath\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_xpath_\u001b[0m\u001b[1;33m)\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     39\u001b[0m                 for e in 主要元素]\n\u001b[0;32m     40\u001b[0m         \u001b[1;32mreturn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m暂存结果\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'e' is not defined"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# C-1 多个页面+多个关键词\n",
    "import time\n",
    "from random import random\n",
    "\n",
    "url = \"https://weixin.sogou.com/weixin\"\n",
    "xpath_翻页a = '//div[@class=\"p-fy\"]/a[starts-with(@href,\"?query\")]'\n",
    "# 多界面（不同的内容（分类、关键词））多页面（翻页）\n",
    "query = ['python','UX','李嘉琪','英雄联盟']\n",
    "list_df = list()\n",
    "\n",
    "## 第一页试探有多长的页面\n",
    "for key in query:\n",
    "    payload = 参数模板生成(query=[key], page=['1'])\n",
    "    print(payload)\n",
    "    df = request_sougou_weixin( url, params = payload)\n",
    "    href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "    df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "    print(df)\n",
    "    df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query']])\n",
    "    df_qs = df_qs.assign (page_int=df_qs.page.astype(int)) # 变成整数\n",
    "    长度 = df_qs.page_int.max()\n",
    "    参数_query_X_page = { \n",
    "        i:参数模板生成(page = [i], \\\n",
    "                      query = [key]) \\\n",
    "        for i in range(df_qs.page_int.min()-1,长度)\\\n",
    "        }\n",
    "    #print (参数_query_X_page)\n",
    "    print (key,长度)\n",
    "    print(\"参数_query_X_page\",参数_query_X_page)\n",
    "    \n",
    "    for k,v in 参数_query_X_page.items():\n",
    "        payload = v\n",
    "        print(\"payload\",payload)\n",
    "        print(url)\n",
    "        df = request_sougou_weixin( url, params = payload)\n",
    "        \n",
    "        time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "        # 备份\n",
    "        display(df)\n",
    "        df.to_csv(\"20春_Web数据挖掘_week05_sougou_weixin_{key}_{k}.tsv\"\\\n",
    "                  .format(key=key, k=k), sep=\"\\t\", encoding=\"utf8\")\n",
    "        \n",
    "        df = df.assign (query = key)  # 区分  keyword    \n",
    "        df = df.assign (page = k)  # 区分  page    \n",
    "        list_df.append(df)\n",
    "        \n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week05_sougou_weixin_翻页_5.xlsx\",\\\n",
    "                sheet_name=\"_\".join(query))\n",
    "# 预估时间: 4*5秒*10 =200\n",
    "# 预估数量: 4*40*10 =1600"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "297.391px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
